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ABSTRACT 


The objective of this study was to investigate a pro- 
posed method of auditing cost-effectiveness analyses of 
technotogiucalilichanges.)) They method as; based on the use) omy cost 
functions to predict costs of operating with and without the 
improved technology. A comparison of these predicted costs 
indicates whether the technological change is cost-effective. 
Results of the comparison can then be used to audit more 
detailed analyses of the change. The investigation employed 
econometric techniques in a case study application of the 
method to a technological change which occurred at the Naval 


Air Rework Facility, North Island, San Diego, California. 
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I. INTRODUCTION 


A. DEFINITION OF THE PROBLEM 

It is commonly accepted that technological advancements 
are arriving at a faster and faster pace with the state of 
the art generally far ahead of the functional application. 
One reason for this gap is the costly enterprise of transfer- 
imgwcheouy into practace. | When the anatiall transter 1s made 
a natural question to ask prior to extensive implementation 
is "will it be cost-effective?" Current methods of cost- 
effectiveness analysis are tedious and usually employ a 
detailed analysis of the process which incorporated the 
change. Such analyses are inherently imprecise due to the 
possibilities of ee dependencies, double counting 
of costs, and external changes which may affect the effective- 
ness of the process and yet not be accounted for in the 
analysis. Thus when a cost-effectiveness analysis is com- 
pleted a reviewing agency might well ask, "Are the conclusions 
of the analysis correct?'*" As mentioned, the methods em- 
ployed in the analysis make this second question a hard one 
to answer. 

This study is solely concerned with investigating a 
method which may simplify the task of answering the second 


question; that is, a method for auditing cost-effectiveness 
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analyses of technological changes. The investigation was 
accomplished by applying the method to a specific technolo- 
gical change which occurred at the Naval Air Rework Facility 
at North Island, California (NARFNI). While an indirect 
product of this study is a statement concerning the cost- 
effectiveness of the change which occurred at NARFNI, it is 
emphasized that the purpose of the study is to investigate 
and refine an auditing method initially proposed by Robert 
iL. Spooner [Ref . Ay, which was based on the estimation of 
production functions through linear least-squares regression 
techniques. A description of Spooner's proposal and later 
refinements established by Charles W. Bradley Ref. af and 


Walduovere (Gq WMeeurorn (AXex0., 6/ will be given in Chapter II. 


B. NARFNI 

NARFNI is one of seven naval air rework facilities 
throughout the continental United States responsible for 
major maintenance, conversion and repair of United States 
Navy and Marine Corps aircraft and related components. To 
accomplish this mission NARFNI employes approximately 6800 
Civilian workers and spends $150 million on annual operating 
expenses [Bradley , pp. 8 and Wye 

Maintenance of equipment is accomplished under one of 
three primary programs, with two cf the programs receiving 
inputs either directly from the customer or from the third 


progam (seembag). i), 
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FIGURE 1. NARFNI Repair Programs 


Under these programs work is further assigned to one of 
the six divisions of the Production Department identified in 


Figure 2. 


Weapons #1 

Weapons #2 
Hydrolic/Mechanical 
Power Plant 

Avionics 

Components and Metal 


FIGURE 2. Production Department Divisions 


The assignment of work to various programs and production 
divisions can be thought of as a "job shop" operation in 
which each shop performs a specific task. Thus an item 
entering ewok is subject to being dismantled, its sub- 
assemblies distributed for rework, and then reassembled prior 
to returning to the customer. This type of operation places 
emphasis on the scheduling of rework in that ideally the 
shops are completing work on related parts in the same se- 


quence to reduce the total repair time of the major end items. 
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Scheduling is also important to the shop managers who must 
order material and plan personnel assignments based on the 
projected arrival lof work from other shops. This entire 
scheduling problem is intimately related to the technological 
change to be described in the following section. 

NARFNI is primarily funded by the Navy Industrial Fund 
(NIF) which provides a working capital fund to finance repair 
operations. The finished product is returned with a bill 
to the customer for the work done. This "debt"! is then paid 
by transfering funds from the customer's appropriated main- 
tenance funds to the NARF's working capital fund. Under this 
concept NARFNI is required to control its finances to incur 
Zero) profit at the end of each fiscal year. This contxol is 
exercised at Quarterly Planning Conferences in which repre- 
sentatives from the customers and NARFNI meet to plan the 
following quarter's work load input and prices for the work 
to be accomplished. 

The work load is measured in number of required man 
hours (NORM) of direct labor needed to accomplish the work 
requested by the customers. The NORM is estimated from 
historical data maintained by NARFNI on each type of work 
it has accomplished in the past. 

NARFNI is reimbursed under basically two types of con- 


tracts, cost reimbursable and fixed price, the majority of 
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the work being performed under fixed price contracts estab- 


lished at the Quarterly Planning Conference. This type of 


contract gives NARFNI incentive to minimize costs subject 


to the required work load, a fact which is instrumental to 


the auditing method investigated. 


C. WORK IN PROCESS INVENTORY CONTROL SYSTEM (WIPICS) 


In January, 1972, NARFNI installed an industrial informa- 


tion system to assist in the location and scheduling of 


inventory in the rework process. WIPICS was requested and 


initiated by the Management Systems Development Office (MSDO) 


in an effort to assist the NARF's in performing their mission 


at a minimum cost to the government. If the prototype at 


NARFNI is determined to be cost-effective WIPICS will then 


be installed at the remaining six NARF's. Its major milestone 


are listed in Figure 3. 


Early 1969 - 
Late 1969 - 
“Early 1970 - 
July 1970 - 
Late 1970 - 


OMANODUAWDN HP 


ROHR Corporation studied NARFNI 
MSDO requested contract 
NAVAIRSYSCOM evaluation and approval 
SECNAV authority for prototype 
Contract negotiated 


4 January 1971 - Contract date, D-Day 
D+ 9.5 months - WIPICS developed 

D + 11 months 
ID IIS Mmoncas = WIPICS jorecovonyjoe Sess! 


- WIPICS test 


FIGURE 3. WIPICS Milestones 
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WIPICS consists of a central computer which is linked 
to the job shops by 164 "touch tone telephones, 20 alpha- 
numeric terminals, and four teletypewriters. In addition, 
the system is equipped with an 80 word audio-response unit 
to allow two-way communication through the telephones. When 
an item first arrives at NARFNI it is broken down into 
identifiable subassemblies. For each of these Gre 
a computer record is created and entered into WIPICS. This 
record contains information such as identification, required 
work, required material, location, and status. Each job 
shop inputs arrival and departure information through the 
"™touch-tone"™ telephones. Also, an unexpected delay in work 
due to lack of material or some other shortcoming is inputed 
as it occurs. At the completion of the work day a print-out 
of this information is produced to aid shop managers in 
projecting the rate and mix of work that will be entering 
their respective shops in the near future. 

The inauguration of the WIPICS prototype was accomplished 
by creating computer records for all jobs presently in the 
shop to preclude a lengthy start up before evaluation could 
begin. Simultaneously, the operational costs accountable a 
WIPICS were transfered as overhead to be applied to the price 
Oh VODSean SOP. ints ducing Jamuary..) O72). NARBNL ) tna tiated 


a technological change which was to be subjected to 
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cost-effectiveness analysis and, at the same time, provided 
a suitable situation to use in investigating a method for 


auditing cost-effectiveness analyses. 


D. SCOPE OF STUDY 
Previous work conducted by Spooner, Bradley, and Trafton 

was directed at completing a methodology for determining 

if NARFNI is operating more efficiently after installation 
of the WIPICS prototype. However, the method is relatively 
new itself and numerous problems prevented completion of 
the effort. One major problem was the lack of production 
data after the incorporation of WIPICS. The purpose of thas 
study is to complete that work. The remaining portions of 
the study are divided into the following catagories: 


1. Discussion of the methodology and production data 
available from NARFNI. 


2. Preliminary estimation of NARF costs as functions 
of the work load and prices of inputs, and analysis 
of the characteristics of these functions. 

3. Estimation and analysis of before and after WIPICS 
cost functions followed by a direct comparison of the 


two cost situations. 


4. Discussion of the exogeneous variables which were 
Wnotlanc lauded an wthevestimation of the cost functions. 
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Ii. METHODOLOGY AND DATA 


A. GENERAL 
The instructional objective is to determine the effective- 
ness of WIPICS. The major difficulty in attaining this 
objective is the probable abundance of other changes which 
make the before and after WIPICS situations different. Thus, 
a direct comparison of costs is invalid. The ideal solution 
would be to operate two ‘tidentical" NARF's, one with and one 
without WIPICS, and then compare costs., While this solution 
is unobtainable an estimation of its outcome is not. The 
costs of those two NARFs can be estimated through the use of 
econometric techniques to construct before and after WIPICS 
cost functions. The estimation methods were reported by 
Trafton but are outlined in Sections B, C, and D for reference. 
In order to estimate the cost functions their structure 
must. first be specified by solving a mathematical program in 
which NARFNI is assumed to be minimizing costs subject to a 
production function, 
minimize C= PjX,t wae es 
(1) 
Sulbgieece auton Yo > Vi(Xj)1-268)): 
As stated in Chapter —, NARFNI has three major programs, 
each producing many different outputs. For example, the 


airframe program produces airframe repairs on 20 quite 
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different aircraft, and the engine program has an even more 
varied output. To obtain a more tractable situation the 
problem at (1) was solved and then applied to the major 

NARFNI programs separately. The result is one cost function 
for the airframe program and one for the engine program. 
Before describing the form of the cost functions a description 
of the data available and the types of production functions 


assumed to hold is necessary. 


B. NARFNI PRODUCTION DATA 
The data was obtained from NARFNI's Production Performance 
Report, a quarterly report which details the total resources 
expended on each airframe and engine which has completed its 
repair process. The report does not include detailed infor- 
mation on the component parts program but rather a monthly 
summary of total resources expended in that program. A 
description of the data for airframes and engines is given 
below. 
alle Identification - includes the type of aircraft or 
engine and the extent of work required, such as 
overhaul, repair ot crash damage, modification, or 
conversion. The type of work required affects the 
NORM assigned to the item. 
2. NORM - is NARFNI's estimate of the direct labor hours 


required to accomplish the work. NORM is established 
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at the Quarterly Planning Conference before in- 
duction of the item. 

3. Induction and completion date - are the Julian dates 
the item actually entered rework and then work 
completed. 

An Da rect labor hows (Bee is the total labor hours 
that can be ae db rnenay identified with rework of 
the item. 

See Dinect mlalbormdolllarns, (US) = 1S ther toralcoct or 
the direct labor expended on the item. 

6e)  Matexslalmdotilans (MS) —"als the! total’ cost of *matecvalis 
that can be specifically identified with rework of 
the item. 

ne Oc neada dollars (OS) — sta prorated portron os 
those NARFNI costs which cannot be identified with 
specific items in rework. 

The data taken from the Production Performance Reports 
are raw in the sense that they represent actual costs per 
item. Bradley wrote a computer program which prorates the 
NORM. DEH, DES, Mi ©) fox) cach atem over the number of days 
the item was in shop. The program then aggregates the pro- 
rated data to obtain daily totals for the engine and airframe 
programs for the period under study. Bradley's program 
generates one additional variable, the number of items in 
rework (I) on each day. Trafton modified these programs 
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to include a penalty cost (PS) chargeable to each item for 
every day it was in the NARF ie. p- 22 This was an effort 
to reflect pipeline costs which may be the costs most sig- 
nificantly altered by WIPICS. These penalty costs are 
especially significant in the component parts program. A 
relatively small part, such as’a generator, may be the prime 
reason an F-4 aircraft is non-operational at its home base. 
Unfortunately, lack of data in the component parts program 
Pee enicd inclusion of that program in this thesis. 

Two variables were constructed from the aggregated data. 


Pil = WW Which aSsla, rough Ssunmzogate for the prace 
L of labor, and 


PI 


which is an average penalty cost per item 
L on a given day. 


Notice that PL will vary with changes in the amount of 
overtime employed as well as a general pay raise. It may 
also vary as the distribution of jobs in the work load 


changes from one level of skill to another. 


Cc. PROMUeMGN FUNCTIONS 

Two structures for the production function in the economic 
problem of Section A were used, the Cobb-Douglas and the 
Constanta last cibymOtgeoul Ss taistutnoOn (CES) pEOduct1ony Lune— 
tions. This was done to compare the ability of each to 
model the relationship between NARFNI's inputs and outputs. 


In each program the output is approximated by NORM produced 
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each day. This approximation is necessary due to the numerous 
types of airframes and engines being repaired. Since NORM 

is measured in the same units for every item it is a simple 
way of measuring production while at the same time reflecting 
the amount of work accomplished by NARFNI each day. 

The Cobb-Douglas production function was assumed to have 
three inputs: 

io ib, Wale joreshee: I2IL 

2D. i WAL jodLeS j2 1 

3. M with price 1 (since material is measured in dollars). 
Thus the production function takes on the form 

InN = % 4%, Ink + %2 InI + %, InM. (2) 
The CES production function was assumed to have two inputs, 
Land I, and takes the form 
aa re 
N = y [sr (ER iL e] 

Oe Ibi S Win tf s ce In jer & (1-6) | (3) 
The Cobb-Douglas production function was selected for its 
useful and convenient application in emperial work. It is 
4 GOSCIAL CASS Cif Uae CBS sUINSELCin shin Wlaeie Ware llshmsle Cuz Ale 
CES production function, as @-»0, is the Cobb-Douglas function. 
Dheneronme, sthe  clastaenty, Of Substitution (6) Of anputs as 
assumed fixed at one in the Cobb-Douglas function and it is 


only assumed to be a constant in the CES function VER p- 2277) 
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The CES production function was included in the study 
specifically because it allows LO) Gliieiteie , sic Wu L Ay uni! 
therefore, should present a more accurate model of NARFNI's 
production relation should this be the case. This advantage 
WSvOmEsete wnowevers, by the tact) that the CES function at (3) 
above is not linear in the parameters and estimation of the 


resulting cost function becomes more difficult. 


D. ESTIMATING THE COST FUNCTIONS 
The following procedures apply to both the airframe and 
engine cost function estimation. For this reason only the 
airframe procedures are presented. | 
1. Cobb-Douglas Cost Function 
The cost function is derived by solving 
mein Cs ILL) se iM ss jeae((a0)) 
i rue [see Oo jo 32/ 


where the dependent variable in the objective function is 


st. N 


derinedmas) DES +t 9M 2 Pe. ihe Wwagrangiran) (eG)! as 
ns ap. 
Ue 2 LE) 2M =e PIG) = AUNSNS ie). (4) 


The lst order conditions are obtained by taking partial 


Gerace SMO (4) )swantenes pect tomlin Mn lk wand jy ihe asesuukes 
are: 
A-l py 
LG) = PL +AKAL IM M& (5) 
-1 
1G. = 1+ VAI Tm ebm (6) 
-1 
We, = PX & pam MM, (7D 
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ics =oNeAT a ue” (8) 
where the 's are random variables associated with errors 
due to imperfect cost minimization by NARFNI. 

These lst order conditions can be solved to obtain 
the inputs as functions of the prices and output. Substitu- 
tion of these functions into the logarithmic form of the 
cost function yields 
iInPL + 


Is = ao see LEN 4 InPI + € (9) 


Crp ty tpt y 
where the error term (€) is assumed to be normally distri- 
buted with mean zero and constant variance. 

Standard linear least-squares regression techniques 
LS. Chap. 3/ were used to estimate the coefficients of the 


variables in (9). The results are reported in Chapters III 


and iV. 
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Again, the cost function is obtained by solving 
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The equation at (10) is not linear in the original 
parameters of the production function and, therefore, cannot 
be estimated by the same procedures used on the Cobb-Douglas 
cost function. A more complex two-stage least-squares re- 
gression technique was used instead. The first) stage Consists 


of a linear least-squares estimation of 


linea) =vadeeb dn(Ge/Pljas+ © (a2) 
where 
are a 
a= (Apt) S and b = =(Qoul 0 


Equation (12) is derived from the first order condi tons Of 
the mathematical program [see DTrakton, p. 18/. The result- 
ing estimate of (EPL) AS wien Sulastawbieec! suete@ 1D) are (1). 
The estimate of D is then substituted into (10) which becomes 
eee in the parameters. Thus, the second stage consists 
of a linear least-squares regression to estimate the co- 
Gu eneires Cie (iO). 

A comparison of the Cobb-Douglas and CES cost functions 
derived is discussed in Chapter III where a preliminary 
analysis of the procedure is reported prior to application 


of the entire auditing methodology. 
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E. DIRECT COMPARISON OF THE COST FUNCTIONS 

To estimate the effect of WIPICS on NARFNI's efficiency 
the cost functions derived in Section D were estimated using 
aggregated production data from a period of time prior to the 
inauguration of WIPICS to obtain before-WIPICS engine and 
airframe cost functions. The procedure was then repeated 
using aggregated production data for a onal after WIPICS 


for an after-WIPICS cost function for each program (Fig. 4). 


Before WIPICS After WIPICS 


ae ea eee 
| 
| 


Airframes D cost function CD cost function 
CES cost function CES cost function 
ee 
Engines CD cost function ieCDvicos fuesunciiaen 


FIGURE 4. Cost Functions Estimated 


To compare the two situations the before and after data 
were consolidated and used in the before and after cost func- 
tions to estimate daily program costs for a given period. By 
averaging the difference between the expected cost with 
WIPICS and the expected cost without WIPICS a mean cost 
differential is predicted for each of the following situations: 


1. NARFNI operation with and without' WIPICS using before 
WIPICS observations. 


2. NARFNI operation with and without WIPICS using after 
WIPICS observations. 
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This procedure is followed for both the Cobb-Douglas and CES 
functions. The computer program constructed for this purpose 


is listed at Appendix A. 


F. DATA 

Production data was obtained on the engine program for 
two-separate periods. The first period covered 1865 engines 
inducted on Julian dates 0140 through 1273. The aggregated 
results obtained from this period is listed in Appendix B 
of Bradley's thesis. The second period covered 521 engines 
inducted on 2003 through 3050 with the aggregated results 
listed at Appendix B of this thesis. Data on the ee a 
program covered 564 airframes inducted on O069 through 2234. 
The aggregated results from Julian date 1001 forward are 
listed in Appendix C. 

The operational cost of WIPICS chargeable to NARFNI 
averages approximately $48,000 per month or $1600 per day. 
NARFNI was not held accountable for these charges until 


January, 1973. 
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III PINS IL IIMUCINUA ROY ANALYSIS 


Before attempting to apply the procedures outlined in 
Chapter II a preliminary analysis was performed to develop 
methods of characterizing the estimation techniques. This 
was accomplished by using a AQO day peuiodton datamtovesita- 
mate cost functions and test for stability and accuracy. 
The resulting proceedures were then included in the final 
analysis of the technological change. The 400 day period 
used in this chapter was well before the WIPICS prototype 
became operational. Therefore, the results obtained should 
reveal some of the problems to look for in the estimation 
of the cost functions, that is, the expected variability of 
the cost function due to events other than the implementation 


of WIPICS. 


A. REGRESSION RESULTS 

Using she procedures outlined in Chapter II Cobb-Douglas 
and CES cost functions were estimated from 300 days of 
aggregated production data. The resulting coefficients are 
presented in Table I. The results of the first stage of the 
estimation procedure for the CES cost function is not pre- 
sented unless the coefficients are not significantly different 
from zero. Ree when multiplied by 100, is the percentage of 


total variation in cost that can be explained by the estimated 
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funetion. SE is the standard error of the estimated cost 
function (in logarithmic form), and the number in parenthesis 
is the standard error of the coefficient just above it. 


Standard errors for leading coefficients are not computed. 


Ee ee eee ara ie 
; | | 
| Function Coefficients SE Ro 
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TABLE I. Before WIPICS Cost Functions (Preliminary) 


1 300 observations from O19] to 12206. 


2 300 observations from 0215 to 1150 
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It should be noted that the dependent variable of the 


CES second stage regression is not I1InC but rather 


€ 
os 
Sig) PIL, SE PECs AL) 


Also, the C representing costs in the Cobb-Douglas function 
is the sum of material, penalty, and labor dollars, where as 
the C in the CES regression is overhead, penalty, and labor 
dollars. These two differences may explain why, in the air- 
frame program, the CES function has a higher R2 than the 
Cobb-Douglas (indicating a better fit) while the Cobb-Douglas 
has a lower standard error (indicating it is more accurate). 
The two functions are thus not comparable in this form so 
different methods of comparison were needed. 

A rough test for the significance of the coefficeints 
in Table I can be made by taking the ration of.the coefficient 
to its standard error. If this ratio is less than two the 
coefficient is not significantly different from zero at 
approximately the 5% level of significance. All coefficients 
in Table [I are significantly different from zero by this 
test and the conclusion can be made that each independent 
variable contributes to the explanation of the dependent 


variable. 


26 


i oo 
Ta 
7 
7 
u ; 

i wa = vt i aoe 7 

. iF 
nokroad)-salyuct-cdad efs ni efeoo Bnineaoiges 9 pi 


as Oitfw ,etellob tetal bas. viisnog ,fabiaias to 
todatl haa ,vslsaney , beachhevo al polepang’s 2 an? 
tin of? ni. ,.vde iisBaie "wen asonoe Te rtih buy oven val 
ait nei? © tedgld «& anit Ho'kt>aub 22D a ual 


Satouol-ddod sif etidw (#24 19% 3sd » oaliaothuk) eats 
~(S342u094 stow €f #2 onitaoibat). sorte baabrate 2 
ow orot 2fady ni alfdatagmco tor aut? sta snebian: 


hehet Otonw moetrsqmos Yo. aehoiltom Jz x 


ataieasltiecqo « to wsonsotiingts off ac% fee? Hove 
frmizhliseo wit 9 petiar ed? ovidat? vd siaw ad pas Tt 
way ow new? sest ef offence wfc? 1X1 rae ta Hidbnsla 
tm i “ 3th vitnsorttiagle Caer aa rotomy 


setiingte Yo Lovet Be ed? \lotaeniae 


ne ’ 7 
4d tet mo<d f: ‘dih vtsnestttnete oie tae 
ineoidedqebhwur Foss Part hain ed neo noOlewiadeae wif bas a 
hitac 1c WoOlsenaloee arte , swtedhivies oldal® 
of dn lay 
» 
- 


~ 


Bee  SDABEEEDYOhs He MODEES) 1(CHOWs BES) 

With the large fluxuation in work load and type at NARFNI 
(see NORM and penalty cost values in Appendices B and C) it 
is probable that the production and cost functions for NARFNI's 
programs shift considerably even though technology is in 
fact not changing. Inherent instability such as this may 
camouflage the shift due to technology and should, therefore, 
be tested for by means of the Chow Test 3; pp. IOAL1LNEZ 
The Chow Test is founded on the general principle that a re- 
gression line will not pass through every data point and that 
data obtained from two different populations can be better 
fit with a regression for each population than with a single 
regression for the total data set. If the converse of the 
above statement is assumed to hold, then a test to see if two 
sets of data are from the same population may be obtained by 
seeing if a single regression fits as well as two separate 
regressions on the respective data sets. To extend this test 
to the cost functions a second regression was performed on 
the 100 day periods following the 300 days in each of the 
two programs under study. The data was then combined and 
regressions run on the 400 day periods. The test statistic 
used was 
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The subscripts refer to the respective data periods of 400, 
300, and 100 days. SE is the resulting equation's standard 
error, n is the number of observations, and K is the number 
of coefficients estimated in each equation. If the test 
statistic is too large then the hypothesis that the two 
periods have the same cost function is rejected. Results are 


presented in Table II. 


Test 
Model Statistic Critical Value Results 
PU R( 2822) 
Airframes | 2neoe | Reject 
In(S, 390) | | 
28.78 2) Pile AOR ReTect || 
[TES 202) 4 
Engines 14.741 re 2.43 Reject | 
| B(S,e50)) | 
ces | 41.22 eel De QT | Reject | 


TABLE II. Chow Test Results (Preliminary) 


The entire 400 day periods used in the Chow Test fell 
well before WIPICS was operational at NARFNI indicating the 
cost Ene acne are not stable, even with fixed technology. 
This revealed a possible problem in applying the auditing 
methodology proposed by Spooner. However, it may be possible 
to dimenish the importance of the instability if it is not 
too great. This can be done by increasing the sample size to 


well over 400 days, thereby allowing the variations in work 


28 


ene. An sbotaog stab ovis rome: 
bishyare etek kupa) sails ‘een 
ary baa. sana Bays9a00 90 


ee 
Jeet oti it Jinetioube dose: nt ‘Seavieia 4) toa 


ows sat? inde ekeoddciggd (ott aon) abel o8t hae aie 
—! | i ew 
je> @%8 pliveed .-batosist el \nottonu® Ip00) gaae oth eVat 


- 4 afdat tw 


tan? 


: etluaeh oulayv banter kxd ilizijate 


(OCE,e)a 


: SoS , 
_ (Sve. be . _ 
f2eben | €b.5 |. bev 
7 (oer ,2)4 ' 
. fost ef | te 8.6 
a {yrscqlwttess) erlvest .treoT wat?  ;TY BIBAT 
. 
_ 
: : Lia fewT weld a wi bees eboriey vab OOF etitage@ 
-_ wif gel ee¢ i Phen | re i Ly ie AraéTaraqo 26 Sori ry aro 


ais 

»Yenicorlisr 6b» '} Atiw lieve , eldata fon @3a enortons 
; 

olvihuia aff Ofiylen ii amEidoiqg efdledng 4 betagveg 
Szzeure ud ved 3t to wool! Tero yo he@egese vigul@sa a 
> ; Pivtda? i at? Yo esos? vocal aty satboows 


4 rz ray ait ii £“e7 >of. ad cack wd flas abhaT TASTY. OF 


CEE! Lv, ' offwolfis vderen? ively COF 19¥0 Lfow 


AS 


—_ 


load to average out. This technique would prove ineffective 
against large, prolonged projects of a particular work type 
such as major modification of all F-4 aircraft in the Navy's 
inventory. In addition, this solution would mean waiting, 
perhaps for years, before obtaining enough data, after the 
technological change, to conduct the analysis. In most cases 
the requirement for the study would have long since been 
dropped. 

A more generally acceptable solution would be to accept 
the cost function for use as long as it can accurately pre- 
dict costs while at the same time keeping the accurracy in 
mind when final conclusions are drawn. This acceptance can 
only be justified if the accuracies of the function are 


measured. 


C. ACCURACY OF THE COST FUNCTIONS i 


i Empixical Confidence antervals 


2 ous } . th 
If e, is the error of the predicted cost for the 1 


OeeCrUAon 4 confidence interval can be empiricly constructed 
by using the ratio of two standard errors? of the function 
over the jvactiali mean Costs hom ithe period.) fhe result, as a 
percent error in cost, 


The term "standard error't is miss-used here since 
nothing is known about the distribution of the error terms. 
Still, the procedure is not uncommon. 
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% error = BAG es x 100. 


The % error approximates + 2 standard deviations which 
accounts for about 90% of the mass in most probability 
distributions. Thus, there is approximately a 90% confidence 
that the predicted cost is within the percent error com- 
puted of the actual cost. 

This procedure was performed on the cost functions esti- 
mated from the 300 day period, and then on the 100 day period 


with costs being predicted by the 300 day models (Table III). 


Model 300 Observatims 100 Observations 


O4 . 0% EOS 076 
SO EER 


Engines O7 .0% ee US ee 
sis TI an ee aor 


WARES IIS Ya ISacseones alia) (COS UinetLems (IPceslalmsingeucy,) 


Airframes 


lplieh ele + 


As the standard errors seemed to indicate in Table I, 
the CES cost function appears better for the airframe pro- 
gram while the results are mixed for the engine program. 
The largest errors are in the engine program when the functions 
were used to predict costs. This is understandable since 
the engine work load and mix of work fluctuate more rapidly 
and extremely than those of the airframe program. The re- 


sults also indicate that the engine cost functions may be 
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highly sensitive to small shifts in the data period used. 
Finally, in the 300 day functions for engines, the Cobb- 
Douglas model appears to do a better job than the CES. This 
is counter to expected results and VSimok) explaamed® 
Bo Statistical Prediction Intervals 

A similar procedure to that of subsection 1 uses 
statistical methods outlined in Theil [pp - 13435) BO COwe 
struct prediction intervals. et C represent the estimated 
cost and accept the assumptions of the standard linear model 
{Bs pp. ARO ILI) Then the following probability statement 
holds: 


P(x = It Ss dine = inc; + H) = 1-4 


where 
Ueto 
H = Coy, (SE) Hons We OS 2) Vo 
X = matrix of independent variablesused to construct 
the model 
Vo = vector of independent variables used to predict 
nC y 


ge { -statistic 
This interval is in logarithms. The transformed interval 
in costs becomes non-symetric and non-linear in the inde- 
pendent variable of Vo. Thus, it is more accurate to compare 
the extreme widths of the intervals than the mean values 


compared in subsection 1. 
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Prediction intervals were computed for the 100 day 
period using the 300 day models of Table I. Of the 100 
prediction intervals resulting the minimum and maximum in- 
terval were transformed to dollars and nent as vi by the 
mean cost for the 100 day period to obtain a maximum and 
minimum width as a percentage of the average cost (parenthe- 
iPAccHmeniaakes som WWab lerm Vie “hanadeliye the percentage of actual 
costs that fell outside their corresponding prediction 
interval is reported in the last column of Table IV. Com- 
puter programs written to perform these computations are 
reproduced at Appendices D and E. 

Max. Min. % Out- 


Model Interval _ Interval side Interval 
5305 4% 


Airframes CD ; (or ) 
CES 
Engines { CD 


TABLE IV. Extreme Widths of 90% Prediction 
Intervals (in dollars) 


3.) Accumacy OreBenMeAsunes 
Table III and IV should be roughly comparable by 
doubling the % error under the 300 observations column of 
Table III. This value should fall between the maximum and 


minimum percentage values for the same function in Table IV. 
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As can be seen, the statistical values of Table IV tend to be 
smaller when this comparison is made. Moreover, a larger 
discrepancy between the two tables is accompanied by a larger 
percentage of the actuaill costs falling outside of the pre- 
GAC Ee HOmMmaeniRer Vac sorts, hat Ker Ia)” 

Some of this difference may be accounted for as 
interpolation errors for the t-statistic used in the equa- 
tLOMm Qe, (7) tO COtestin MALS WWs JBsesceKeS slo Woks Ses Sibsle  eucs 
exponential in nature and the resulting Seehiots om intervals 
were found to be quite sensitive to the value used. The same 

-statistic value was used for all of the functions in 
Table IV which seems to imply the value chosen was low. A 
slight increase in this value would increase the prediction 
intervals, make Table IV more consistent with Table III, 
and should, at the same time, decrease the percentage of 
actual costs falling outside the corresponding prediction 
interval. More care was taken in selecting this value in 
Chapter IV where the auditing methodology is carried through 
completely. 

Finally, the uneven prediction results between the 
cost functions of Table IV may be indicative of the Chow 
Test's conclusion that the two periods of data, 300 vs. the 
following 100 days, are in fact different and a model of one 


period is a poor prediction of costs in another period. This 


33 


oa or ol vr 1 asin pte 


TITEL saevesicle: aabaa Pree: ening Akad ee 
erent acy) balosgemans 5% asldat wont ad pent se 
' a nile 2G obkenie. waked etaog abe, sushi 
sue haat aa aves ; 
#6 10% .be ces ml yeu Gano«wsvh ih eis to ‘ | 
sips, ant pi beau uttelighn= +—ent Jorn ro3 ks noital : 
ota obteitoty aidd wk aio13a V1 aida aint Qt Any? 


a 


aiaysetdaicolteibeiq prtsiteder std hina aohrac ni la 
4 
gaae s1T .boew solav of2-62s oviiienss @tiup od: oF ber 
nmb‘enotionwt at Yo Ile tot beeu aaw\eukay Sie 

A. .wol saw reads bulav wd? vlan! of emeen do tite 
no tTolihbetd et! e@asisal bivow sulav ekAT ol sanert 
elit oidet tie tnetetenoo sxyom VI eldat atau 42 

to spatresseq at? eusaxooly mi? seep of ta, Of 
ioltotbery yathnegeutigo af? sbtetvo gaiilat e1aan om 

it geieav @f8) onliselen Al awta? Zan 4405 wiOW _ 

tpi r bocs7i 22 youloboriam onltibve edt sgedw Vi 
ulated 

ud asewted efluaw lisibetqg nevernw ony ,vitanrt 

1 «7 YO ovilasibal ed “am VI ldet to vpuni frond SB 

OE 4b 30 abedtoeq ows off Dats aAeteuiontes, 


Gre toorettto. souk of exe: eye OOl galwelien 


oitaq iffons of etaoa to noftokbotg Weg h Bh Beraaq 


Ee 


— 


possibility is important because the periods are not separated 
by a recognizable technological change. It reemphasizes the 
need to include the accuracy of the models in the final com- 


parison of estimated costs. 
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IV. COMPARISON OF BEFORE AND AFTER 


WIPICS PREDICTED COSTS 


A. THE COST FUNCTIONS 

The final cost functions used to compare estimated costs 
across the technological change were obtained using the 
SNAP/IEDA Computing Package developed by the Department of 
Statistves Princeton University. «SNAP/E DAY wall accept 
up to 400 observations. In those models requiring over 400 
observations the BIMED-O2R program, developed by the Health 
Sciences Computing Facility, University of California, was 
used for the Cobb-Douglas estimation. The CES estimation 
was limited to 400 observations in all cases since estima- 
tion using the BIMED-O2R program would become a 3 stage 
procedure requiring excessive manipulation and_inefficient 
USS OH Comowieeie wale > ALSO, show Unie ioliedl COSe aibiaKo eels 5 
the data periods were altered for two reasons. First, the 
ae ee were chosen close to the technological change 
so the before and after situations would be as similar as 
possible, except in technology. Second, the data periods 
were altered in size and content to reduce the effects of 
instability as much as possible. The results are presented 


in Tables V and VI with footnotes indicating the data used. 
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Note that the CES function for engines across WIPICS 
was not ees due to the large number of observations 
involved. Omission of this function merely precludes per- 
forming a Chow test which in all probability would be rejected 
if performed. Visual comparison of the before and after 


functions supports this. 


PUUEE ELON Coefficients SE 

CD fF -297 6857 2356 -.094 O14 
Before ( SOL aL) ( .090) (o@2L5) 
WIPICS 


P.999 886 10.220. .025 
| 


Ua ae pas 4.222 ASG, 


.O17 


After | ( 607 i)) (16284) VC 4OSS)) | 
WIPICS 

ZAOMS  >sSseul -2.724 O06 
(LOM) 2 (298) a 
.934 -884 :074 -.179 | 020 

Across6 (.017) @035)) (6023) } 

WIPICS | 
~945_ -907 -~3.424 .O31 

( .025) Ge S18) i 

ABE Vey Actaatidames Cos em Mune ta OS 

4 


400 observations from 0316 to 1350. 


> 60) cheexvations ram DOGO ko 27. 


Peritodsaa vance 2 gabove lexcepundatesmencing im 34) 5). om!i7, 
deleted to reduce the number of observations to 343. This tech- 
nique was applied to the before and after case with essentially 
the same coefficients resulting as reported above. Thus, de- 
letion of some data should not have Significantly aineecied the 
across WIPICS coefficients. 
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463 observations from 


400 observations from 


160 observations from 


Function Coefficients dase) SE R 
Before eo | DAO 837 147% 494 O51 | .819| 
WIPICS (.018) (3201) (.058) | 
i i 
ces” 1.987 897 -.135* O59 | 4755) 
| ! (O26) 0 4222) | 
eG os as So 
Miter 3 1G |) 4.02 708 = 50 nels 1029 922 
WIPICS | | ( .020) Genle3}) (.080) | | 
iCES | “134,11  ,Ag7 =,812 035 | 861 
i (Gone) 327) 
MEU sesaatnes Auatiane | Tans! tn chlwatnsiaa 
| | | 
Across, lcD 4) 35408 52D) 345 2g OS5Oi4 3854 
Wiptes: | (foie) (043) (ose) || 
i { i ! 
i, t 
{CES i not estimated j 
PM eae! | 
TABLE VI Engine Cost Functions 
* Not significant at the 5% level. 
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All coefficients for the airframe functions are signi- 
ficantly different from zero. In addition to the two starred 
coefficients which were not Significant in the engine functions, 
TNS COSieieal online (oe. Ike) GAB) in the first stage of the before 
WIPICS CES estimation was not significant. Since the 
functions were estimated from varying time periods the SE 
and Ra are representative of only the Pocebet ive function and 
should not be compared across functions. Thus, to characterize 
the functions the Chow Test and statistical prediction inter- 


val discussed in Chapter III were used. 


B. STABILITY ACROSS THE CHANGE 

The Chow Test was used on the cost functions to determine 
if they were in fact different before and after WIPICS 
(Table VII). 


Crattarcall! Results 
Model Test Statistic Value 


Airframes dD), 


16.94 Reject 


CES Reject 
CD 
Engines 


(Not computed) 


TABLE VIL. Chow Test Results 
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In all tests the hypothesis that the before and after WIPICS's 
cost functions are the same was soundly rejected. This im- 
plies the additional conclusion that a technological change 
tmay'"' have affected the relationship between imputs and out- 


puts. 


C. PREDICTION INTERVALS 

90% prediction intervals were calculated for the before 
and after WIPICS's functions by using the functions to pre- 
dict costs for the last 100 days (90 days for after WIPICS 
airframe functions) of this respective time periods. The 
same procedure used to construct Table IV was then used 6 
construct Table VIII. 

The most striking result is the improvement obtained in 
percent of observations falling outside the prediction in- 
terval. This occurred because the 100 costs predicted were 
EN ACE WSU Sel On athe) Samp ke used tolconstruct the cost 
functions. A second difference from Table IV is the size 
of the prediction intervals. The change was mixed, some 
larger and some smaller. This again points to the relatively 
unstable situation at NARFNI in which prices and amounts of 
inputs appear to be varying widely. 

Although the prediction intervals are non-symetric about 
the predicted cost, one half of ene maximum interval should 


Safely represent at least two standard errors of the cost 
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FUNCT On Mons Value waliibewwSed) to) male) a) (sagnulia cance 
statement about the cost comparison of operating with and 


without WIPICS. 


D. DIRECT COMPARISON OF PREDICTED COSTS 
Following the comparison procedures discussed in Section 
E, Chapter II, the cost differentials in Tables IX and X 


were obtained. 


CD CES 
BW-Aw) i; ( BW-AW) 


Before WIPICS §17885 § 3306 
Data 


After WIPICS $7129 


Data 


TABLE IX. Difference Between Predicted 
Costs” - Airframes 


11 positive value indicates operation with WIPICS is 
estimated to be cheaper. 
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Before WIPICS 
Data 


After WIPICS 


Data 
TABLE X. Difference Between Predicted 
Costsi2 - Engines 
Table IX is read as saying, ‘Based on the before and 


after WIPICS Cobb-Douglas cost functions it is estimated 
that WIPICS could have saved NARFNI $17,885 per day in the 
airframe program during the period 0316 to 1350, had WPICS 
been in operation at that time.'' Notice that this says nothing 
about changes which these cost anee isan cannot measure, 
such as an arbitrary change in the NORMS assigned to in 
coming repair items. These factors will be We lea in 
(Cimeyoreese WG bia addition, the statement does not consider 

the inaccuracies of the models which were measured in Table 
VIII. It should be qualified by the assumption that the 

, difference is not significant unless it exceeds, in absolute 


value, the sum of one-half of the two maximum prediction 


12 positive value indicates operation with WIPICS is 


estimated to be cheaper. 
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intervals of the cost functions involved. In the above ex- 
ample this sum is $4776 (computed from Walle WALI), Wiis, 
the statement is significant with at least 90% confidence 
(asta, disregarding externalities to be discussed aia) (Chaieyon@ie 
W)) 

Using the above procedure to interpret Tables IX and X 
the CD models indicate a significant advantage in the azr- 
frame program and an insignificant disadvantage in the engine 
program. The CES models indicate an insignificant WIPICS 
advantage in both programs. 

The above comparison could be refined by removing the 
effects of externalities. This should reduce the total 
variance of costs and it should reduce the maximum predic- 
tion intervals of the cost functions. The net result would 
then be a smaller region in which the cost differential would 
be considered insignificant, but also possibly smaller in- 


dicated cost savings. 
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V. VARIABLES EXOGENEOUS TO WIPICS 


The Cobb-Douglas and CES models do an imperfect job of 
SSEMMABIING) TAS SiwricsSere Oak WAVES, Ojo) jOneOCWSEACio GOSS Cue co 
many changes within the periods which were not caused by 
WIPICS. If the major changes were isolated, and the data 
corrected to account for them, then the proceedures discussed 
in this thesis should give an acceptable estimation of WIPICS'! 


effect. Some of these changes are noted below. 


A. PRICE CHANGES 

Looking at the minimization probiem of Chapter II, an 
increase in the price of an input would result in higher 
costs to produce a given level of output. Alternatively, 
a cost function estimated from data with low prices would 
tend to underestimate the cost of production for data taken 
When prices are high. For this reason increasing prices would 
depreciate the eee. effect of a technological change. The 
jE) Ope diceet labor and penalty costs are accounted for in 
the cost functions used for this study so their fluctuations 
should not affect the final comparison of predicted costs. 
This is not the case for material prices and the prices of 
indirect labor absorbed into overhead. Shifts in these 
latter prices could, and should, be corrected by the appli- 


CAuwTLO9! (Ope “By josaale~) piavoles<- 
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B. COMMAND POLICY CHANGES 

This area is often elusive and difficult to identify with 
specific actions. However, obvious changes, such as a new 
ceiling on allowable overtime, should be accounted for. 

An increase in overtime, if not a result of command policy 
changes, may be caused by temporary inefficiency and shop 
conjestion due to the learning process that often follows a 
technological change. In this case the correction would 
consist of identifying the period affected and deleting it 
from the study. 

Another example is a shift in the standards for assign- 
ing NORMS at NARFNI. The shift is acceptable if experience 
shows that required labor hours have actually changed. Any 
other reason for the shift would result in an imaginary shift 
in the production function with a concurrent change in the 
derived cost function. This type of bias could also be 
corrected by application of an index to the data before 


estimating the cost functions. 


C. PERSONNEL TURNOVER 

A season of large personnel cuts followed by hiring of 
new personnel would naturally establish a training period 
sLig} INVAVRORINIE ,  WDYwITEaLIgG) TEIMGLSS jovszcsloicl) Wins reSleyieswe SsesesifealSinichy Od 
workers would be low and a corresponding change in the cost 


FIDeCELOn WOuld be moted. lt the traanang occurred during a 
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period used to estimate a cost function the function would 
inherit a bias resulting in comparison errors. The effect 

is complex, however, it may be that wages paid to personnel 
are low enough, relatively speaking, to allow the cost of 
production at a given level to remain approximately constant. 
Still, the relation between the fete would be altered and 


a detailed study would be required to correct for the bias. 


D. CHANGES IN TYPE OF WORK REQUIRED 

In Chapter I it was explained that NARFNI has many different 
outputs in the form of repairs to various types of airframes 
and engines. These numerous outputs were aggregated into 
only two products, repairs of airframes and engines. The 
cost functions associated with these two products should do 
an acceptable job of predicting costs so long as stella yaTmallsca @ste 
work remains fairly constant within the respective programs. 
A shift in work type cannot be accounted for by the cost 
functions. Unfortunately, there is strong evidence that the 
work loads are shifting considerably. An approach to account - 
ing for these shifts ic discussed in Section C of Chapter VI, 


PROBLEMS OF FURTHER RESEARCH. 
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VI. CONCLUSIONS AND PROBLEMS OF FURTHER RESEARCH 


A. EFFECTIVENESS OF WIPICS 

The comparison of estimated costs for the "with" and 
“without'" WIPICS situations indicated a savings in the air- 
frame program and a loss in the engine program. The results 
of the two program comparisons are not additive due to the 
difference in time periods considered in each program. In 
addition, the component parts program was excluded from the 
study because of data format. For these reasons, an overall 
conclusion concerning the cost-effectiveness of WIPICS could 
not be extracted from this study. 

Upon completion of the study it was found that WIPICS 
had not been fully operational in the engine and airframe 
programs for the periods originally stated. Numerous 
oxoplens in implementing the system prevented a complete test 
of the system during this period. WIPICS has been completely 
operational in the airframe program since October of 1972 
but was not operational in the engine program as of 4 May, 1973. 

The indication of cnet Savings in the airframe 
program may be explained by shifts in the external variables 
discussed in Chapter V, a prime suspect being changes in the 


types of work. 
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B. AUDITING METHODOLOGY 

Investigation of the auditing methodology originally 
proposed by Spooner revealed eight primary steps that are 
SSSSmMETMEIL TO Eucislwaline} ENB EY Sicic@inc) oine liuissloym 3 

1. Identify external variations such as command policies 
and inflation of prices so they may be accounted for 
in the data. 

2. Correct the data for the variations identified in 
iNebovesombhe ysituat ons, betrorevand iaiter abe 
technological change will be as similar as possible. 

3. Aggregate the data. 

4. Select the periods to study, keeping in mind that 
closeness to the change will aid in stabilizing 
functions while, at the same time, a delay may be 


needed to allow the technological change to become 


noticeable. 
5. Estimate the cost functions. 
6. Estimate the accurracy of the functions. 
7. Compare estimated costs derived from the cost functions. 
8. Draw conclusions from 6 and 7 above. 


The audit conclusions may be more meaningful if identical 
time periods are used in each program studied (assuming there 
is more than one). This will allow the separate effects to 
be additive for an overall statement about the effectiveness 


of the change. 
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Both the Cobb-Douglas and CES cost functions appeared to 
vary in ability to represent the production relation of inputs 
to outputs. Neither function dominated the other in accuracy 
therefore no conclusion can be made as to which should be 
used. If only one is to be used the natural choice is the 
Simpler Cobb-Douglas function. 

The results of the WIPICS test case indicate there remain 
problems with the accuracy of the tests. It is believed that 
Telativelaysample treatment om the datal som external analu= 


ences would improve this accuracy. 


€. PROBLEMS OF FURTHER RESEARCH 

There are three primary areas in the auditing of cost- 
effectiveness analyses undergoing current research at the 
Naval Postgraduate School. 

First, Professor James Hartman is applying search tech- 
MLGueS~an ani attempt §O improve the estimation of the 
panaMetexncmins he ICES ucost sunctlons.) ihe 2-stage wleast— 
aes regression used in this thesis results in two estimates 
OF Ele SLAG Ey Cle GUIS A wwIE OI) SLjo)’ IlAey joreoMSiestOral  sebYgOeaLoyo| , 
both of which are significant. The problem is that the two 
estimates are invariably quite different. It is hoped that 
better estimates of this parameter will result in a more 


accurate CES cost function for use in the auditing methodology. 
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Second, Professor Norman K. Womer is investigating 
methods of accounting for the shifting of work type in the 
various programs which is believed to be responsible for much 
OP themtnaccumacies Anythe models used an thas thesws. fhe 
PLAanci pal /emrout in thas vanea as towaxnd developing a com- 
puter program to perform the regressions on raw data instead 
of the aggregated data. Dummy variables are being employed, 
one for each type of airframe or engine, to account for 
shifting work type. This requires a number of variables 
exceeding the capabilities of the computer soft-ware presently 
available. 

Finally, Captain Richard McGarrahan, USA, is investigating 
a second auditing methodology proposed by Spooner which 
employs a linear input-output production function instead 
of the continuous production functions assumed in this thesis. 
The objective of the work is to establish a basis by which to 
judge the abilities of the continuous models. The input- 
output model is eee ned to be more accurate in that it treats 
each type of work independently and should therefore be more 
stable. If the continuous models compare favorably in 
accuracy they would then be considered more desirable for 


their ease in handling. 
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